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Abstract 



The Dniv^tsltj of Vashi&gton mrnrlrtee on iii^xuctlOQal developKat 
4Ukd innoTttlon iM erralxxkted xrLth respect to tbdr effectlvenen as a 
grmtisg agency* Ibe paper, begiits vlth jt aectloa deflpn^ eTaluatloa and 
then applies that definition in a conceppial 22^ empiticul critique of 
funded activity. Condttslons drawn in^^ae jTO>ort li ^ l od e; (i) evaloation 
ahottld not be required of one-tiae funding for ne^ coarse developocnt^ (2) 
the coadLttee should provide the gr^t receiverVvlth specific evaluation 
re^ulrezaents, (3) the coaoittee should acc^t yue re^misibility for av^** 
-uatlng thB c^^erall ftxndlag pattern ^hhey esta)alsb^ (4) ^ conceptual bast 
of aedla-releted e3^aluatlon neaeds to be ch^mgtd, and (5) funding for the 
iaproveaent of existing caxsrses should c^rtend through aore than one grant* 
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IraiaatiM of tbe Dal v e ral ty of Bfcsiiiaigbra > 

« 

• - * 

...everythlrtg excellent 15 &s dlfftcsilt *s ;lt i* raarc.* 
^ • .* Spiaora f 

of ttfcshlagtOQ h&s bees stmcrcreB tc allov the cotn±ttae laflarrr^ over 

lastrxj^tioa ;3fa&lit7 vl^ judicious control cf fitifiiicltl cwards to te&chen. The 
IjjC^of nonetary influence has often been atated &s a serious handicap in tbe 
•e^rch ^cr in p rov e d teaching, thus the innovative fend prc^grta provides sn ex- 
cellent op^rcxnlty t3 pxfnrirte the jaerit of tbe control of «w»x4» concept* Thl« 
report is coaoemed (IJj^ vith providing a cridqoc of tbe evaloatloa activity 
, associated vlth* fxnded projecte, (2) vith reporting In »3snar7 £om the results 
of soae Edncadoaal Assesaarot Center (EAT) evaluation efforts, and (3} ukins 
recoaoendatloas to be considered by the fund coKittee In fatorer resource- 
allocation decisions > All of tills indirectly assesses the usefulness of the 
control of awards idea. It is our sincere hope that. the report vill contribute 

■ • / ■ 

to strengtiiening tike ispact of the fund snd hence to la^rovlng the quality of 
student- learning at the Qniversity. 

The pap^ is divided into four cajor sections: (1) Vhat is Svaloacion?» 
(2) A Conceptual trltique s>l fwid Evaluation^ Espirical Besults, and 
(4) KecoKendatlons for Future Fundings Tbe section discussing evaluation is 
included because^ vhile oost people are convinced that evaluation is necessasyt 
and» In fact» engsse in evaluation activity^ not xsany people can explicitly 
describe the evaluation process* The literature on educational evaluation si^^ly 
sssirars that everyone understands tbe oeaning of its central tera^ value, and 
gallops into discussion of hew evaluation is to be done* Tbe result of 

4 ' ■■ ■ 



neglectlcg the ccscepltial caderplrmlrtgt of rralutttofi hw beco a sttaatloa in 

vhlch rTcryone says we ostgbt to hsve soiae evalos-tioa (ve agree, opposing craLc*- 

tloa a» a co:ic^t i$ as bad as opposing cothcrhood), bat oo one is really «r« 

trisat it ti that Irvssited. The section oa evaluatioa seeks to ciar$y tbe 

nature of valuatlSQ ^nd to use the concept of value adopted as a fraae of 

f 

'refe rence tot the other three sectioM. of the peper. Tbe definition tn 

/ ' 

this paper idfctifies valuation as a conceptual aad logical activity vhlch 
leads natuTr^ly into the second section of the p^>er, A Conceptual Critique 
of ?und Equation. i«Mle evaluation should not be confused vlth ciEpirical 
actbod, ^ does drax? heavily upon expertmental techniques, hence the third 
^ section^ vhlch reports xhe ?^ults of esspizic^ wrk conducted by SiC« Because 
evaluations conducted by S&C of nonfonded courses have t>earlng on fuaded courses, 
the^-€S5>irlcal section cites a rattge of tAC studies, funded and nonfunded. The 
section contains a suaoary of the observations froQ the conceptual and 

TiC^lrlcal analysis in recoacncndation form. 

/ 

What is Evaluation? 

Philosopher* of logic frequently rely on tvo tenas, intension and extension, 
in describing the use of concepts. The extension of a conc^cpt is coc^sed of 
^ collectlxm of Items, a group of abstract ideas, or a set ofcvents which fall 
^Mlthln the field of applicability of the concept. Intensions are established 
by deflnlxtg the concept in question with word^x^ syabols. Different inten- 
sions nay have the saae extension. For PTsrrple, in a particular rooa of twenty 
people, the intensions established with ''the wocan in tbe red dress ' and *Sall7's 
■othef is the only wocan wearing a red dresu. In this contejct the two Intensifies 
share the saae extension/ in other contexts they would xiot* In logic, thr\ 



3* 

elebcratloi cf ertfaelos leads tc set thetry asd s-erves es the lo-gltai fs?asdatloa 
of ffi&theaitlcfi (Laager, 1953) , tie elsbozax±az. of lawssioa leads to a theory of 
value (Earnsaa, 1967), 

f ^ • 

^ In order to use a rent .precisely, it is aecessary lo give the tent e formal 
dftflqf dm. Eaj^jLsn (1$£7) iias pni;!b^ed tie fcjicviiig definitlim cf •'gooaaeES^'** 
"...a rh1ng I is s gac-d C if .staTooly if (i) I is a C, (2) C has 
th€ intension * c;caslsti:Lg of predicates (3) Z has all the 
;^roperties corresponding cc c." p.- 154. ^— * 



Tbc definition is not nearly as amiaous as it first appears. For ezasiple, ia 

order for object 2 to be a good car, ir'inast first of all be a car (!)• Tat 

object will need to have four v^ieels, be self propelled, etc. Secondly, there 

ccxst be a set of properties vtiicB define the intension of C, the conc^t car (2)« 

is a sy3ii>ol naming that set. We have then a list of predicates (^) vhixh smre 

« 

as elements of the intensicaal eet. Otzr cosplete concept of a car tsay state that 
a car gets **30 izilee to tie gallon of gas*", ^'carries fouf passengers'', '*is inexpen- 
sive to repair", ""costs less than etc. Each of these characteristics is 
part of the predicate liit and is represented by the jsrariable ^. Finally X viU 
be a good C .if it has each of the properties listed. Ihider this intension for car, 
a Honda is i good car vfaile a Cadillac is liot. ^ ' 

"A thing is good insofar as Ix ezenplif ies its concept . This oeans 
that (I) the thing has a certain naze, (2) this narrr has a ceanlng 
defined by a set of properties, and (3) the thing possesses all of the 
properties contained in the izeaning of the naoe.*" (Eartisan, 1967, p* 103) 

I The definition of a good C has a nus&er of ii^rtant iirplications « 

" • . . a difference had to"^ te isade between definitional properties 
and expositional properties. The definitional properties East in 
all cases be fulfilled, the value differentiations adhere to the 
expositional properties exclusively." (Eartcan, 1967, p. 178) 
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If we eater a third object v^iicb ve call a car (deflnltlcnal properties fulfilled) 
Intc ccsaslderatlcn, ve vlll dlff ereatiate its valae with respect tjo tne other tvo 
by asiag the exposltloaal properties (gas iE±leage, TTTTrnSer of passen^n, repair 
expca&es, cost, etc.). Tnli? aev car Is a *'falr** car If It has soias qf the expoti- 
tiooal properties, is a> **iad^ car \i it- lacks ^lone of the properties, and la a . 
"qo good" car if it lacks all the e:5:po8itioaal properties. Value aaal?sis is 
aee ^ becajse nrrrlng is iiz5>reci&e, the concept of car has too luany objects in 
its field zt applicability, r^e^er^dif ferestiatioa is aeeded so ve raak objects 
by laclttding exp>ositional properties vith deftaltioaal properties. lo addlttoa 
to "good", "fair", ^'bad*'^ and '"no good", Earcniaa's approach alLovs us to dtfiae 
"better" • 

"Z is a better C than Y ineaiis that X has laore expositional properties 
of C than ? and £s therefore *tDore of a C than Y. 'Setter than* in 
other vords, relates two oec&ers of the sas^ class, the first of vhich 
has nore of the daas properties than the second." <Harts2n, 1^6r7» p. 162) 
Iha deacriptive capability of enpiiTical research %z used to establish the pro* 
perties of Z and Y. Ih^ exact' content of the list of ezposition&l properties 
cannot be, boverTer, enpirically determined. ^ 

A second icplication.of Hartron's definition is that an object is both good 
asxd bad, valuable or not valuable, depending iq>on the concept being used* An 
object good under one concept ciay certainly be bad under anoAer. Since concepts 
vary» value laxst not^e taken as given in the object but rather rust be seen 

created by the person or persons holding the concept. Value is always value to *^ 

♦ 

aoaeone# 



The third lirplicatloa ts that ' •••valoe ±s ratlx^aal, I can valae a thing 
only if knov tt, that is, if I faiov^it^ naii.e and its properties- ' (Sartsnan, 
' 1^7, p. 109). For this reason ve call In an expert vhca we .are having val« 
problem. Be knovs ©are about the thing than ve do, hence is tsore able to valxje 
it. Knovledge and valuation are thus inrinarely related. (Bote 1: iTnen ve 
apeak of people's motivations ve often r^er to their values as if they vera 
central to iotivaticn, this coscnon rationale sees values as distinct froa 
thinking. EartEsan's definition removes this distinction. Bote 2: Horal philo- 
sophy is now heavily influenced by the writing of John SawU. He has equated 
goodness and rationality (See chapter VII of A Theory of Justice ) without drawing 
upon Bar tsan'^ analysis. ) 

In the writing which follows this definitional section, there are tvD levels 
of evaluation, (1) evaluation of particular Instructio n a l settings and (2) 
evaluation of the particular evaluations- The instructors given innovative 
grants filed repo^t-s evaluating the results of their laproKreaent efforts, those 
reports are in turn evaluated In this paper. Valuation Is refcxlve in nature; 
it Is reasonable to speak of evaluating an evaluation (there Is no theoretical 
reason to stop at just two levels, evaluation is an infinite regress). A given 
, concept with intension *ik vith its corresponding properties (4)j can be used to 
value an object selected by its definitional properties, ffe oay^ in turn ask 
whether that concept is the appropriate concept for use in valuing the object; 
-to do so we need another concept with intension $2 ^ value the first concept. 
The valuing concept at one level of analysis becomes fact for another concept at 
a second level of analysis. (See Aiideraon, 1975c, for further discussion of the' 
structure of evaluation.) '\ 

The Greek "polls'* was an organl^tlon of peop?^ who gathered in order to 



C4ft&bllsh naitual influence (Areadt, 195^). Tae polls Deiber vas e3q>ected to put 
forth his views an the course of action the city state should adopt and to jBupf>ort 
those vlevs vith reasoned arguinent. -In niany cases the aeribers of the polls 
created reason by agreeing to a particular point of view. The political role 
derived froca the polls Is an action role the politician enters an action arena 
vbere he art 0 influence the decisions of a group of siailar politicians. 

The concept of evaluation thus far advanced is clearly related to political 
activity. In chocstng a thought strticture which can be applied to a real setting, 
i.e., the project being evaluated, the evalxiator is entering into tae action arena 
surrounding that setting. The trappings of science, such as statistical and 
research siethodclogy, trat the evalua^^w brings with him should not )^ad us to 
believe be is an objectJLve, unbia^jed participant in the setting. Scientific ap- 
proaches can soBetizaes establish which of two instructional methods leads to 
b,ettjBr perforaance on a test, but such approaches cannot deternine whether that 
is the appropriate test or whether an alternative should be considered. Science 
can help us establish what is happening, but it can't help decide what should be 
happening. However, since isost pro^rans are designed to produce real world con- 
sequences, the evaluator stands with one foot in the pplitlcal realm and one foot 
in the empirical realm. He is concerned with the structure of thought, i.e., the 
jastlf Icatlon, associated with events he ^seeks to evaluate and he is concerned 
with empirical statements about those events. Valuation occurs when he brings the 
expositional properties of the thought structure together with his observations. 

The foregoing discussion of evaluation frames the next two sections, A 

• * * 

Conceptual Critique of Fund Activity and Empirical Results. TTje conceptual 

critique is our entry into the overall action arena of the fund; the empirical 

results report our entry into— the workings of specific cpurses. 

' ' . 9 



A Conceptual Critique of Putxi Evaluation 
Fund ic^ct, froa a strict developaent point*of-view, has been substantial. 
Rev courses are present on corpus, new workbooks and texts are in evidenceT""^^ 
considerable laedia development has occurred. Host of the activity contracted for 
by the proposal writers has been carried out; while soae of this work would have 
been done^y hi^^ily ootivated teachers without fund assistance, the existence of 
the fund has probably promoted interest in course development projects. The 
question that reaains is, "What is the value of these activities promoted by the 
fund?" 

There are two approaches to the valuation of development activity. The lElrst 
of these, which we will call course iaprovement evaluation , ijivolves the esta- 
Jblishaent of '^better than" in the sense of Hartnian's definition (p. 4). If, for 
example, a conversion from traditional instruction to self-paced instruction is 
planned and the content of the course is not being changed, then student achieve- 

nenp may be the measure of choice and evaluation is clearly addressing a "better 

• *• « 

than" question. Student achievement should be assejsSed both in the traditional 

/ 

course and in the changed course with a measure^ consnon to the two courses. It 
Is the conmion content and the cotanon measure which establishes the membership of 
both courses in the same conceptual class and hence makes then comparable under 
the common expositional properties that ^rtman identifies as central to the 
"better than" type of evaluation. A. sedDfid, independent and Important measure 
is assessment of student attitude toward instruction via student ratings. This 
information can also be collected before and after change. 

Most of the proposals which involved attempts to Improve existing copses ^ 
did not Include provision for before change and after change assessment of 



achievement. The failxire to collect data appropriate to evaluation of student 

10 



achleveaent and sWdent attitude is a serious problem. The usual result In 
educational research, when variables thought to Influence instruction are rigor- 
ously exanined, is no significant effect (Dubln & Taveggla, 1968; Getzels & 
Jackson> 1963; Stephens, 1967. Wallen & Travers, 1963), TJie committee would be 
well advised to assuaie that dei^elopaent efforts funded by the comnittee will not 
rc^lt in^reai Ijnprovements and thus plac^ the burden'of proof for positive jre- 
sults squarely upon the grantee* 

The policy statements of the conmittee include one asking for "clearly 
delineated plans for the evaluation .of the project." This policy statement is 
particularly Important since justification of fund activity depends upon the 
contents of the grantees' evaluations. Unless funded, activities which propose 
to improve existing courses do indeed show course Improvement, the fund itself , 
cannot receive a favorable evaluation. The evaluation sections of most of the 
proposals are vague and usxially do not focus attention on stude^ performance as 
strongly as they should. The weakness of the proposal evaluation sections is 
not surprising as most of the grant writers are not used to doing educational 
evaluation. The committee has then the problem of educating its grant writera 
with respect to evaluation and should seek to remedy this problem by providing 
prospec'tive grant writers with^a pamphlet" explaining tTie ne^ for evaluatiotu 
suggesting procedures for evaluation, and finally, providing es^ample grant eval- 
uation sections. The potential grant writer should also^^receive copies of re- 
ports which meet the standards of evaluation expected bjr the committee. (Part 
of the efforts of EAC have be^en directed .toward the development of techniques 
Individual instructors can use to evaluate their own instructional chiinges. See 
Anderson, 1975 (b,c), plus reports soon to be issued. The educational research 
llt«atufe is vealc where single cl aasruum tegeafch techniques are concerned.) 

.11 . . , ' 

■■ ^ '■ ^ 



Given the dismal record of efforts at educational Isprovement reported in the 
education literature, the fund conzalttee must insist upon a strong evaluation 
program if they are to justify the continued existence of the fund in the in- 
struct ional improvement aspect of development. 

After having read the reports coming from the recipients of funds^ we can 
but mourn opportunities lost. A number of the proposals could have bejen struc- 
tured to permit, at the very least, quasi-experimental (Campbell & Stanley, 1963) 
teBVs of the changes implemented in the funded course. In the proposals and in 
the reports, the position was often taken that the merit of a change was beyond 
doubt wher^s in truth the causes for doubt are legion and the empirical evidence 
is lean <See the empirical section of this paper), ^ 

The second approach to e^valuation of development activity is less empirical 
and more conceptual; we will call it. achievement pattern evaluation . The decision 
to add previously untaught courses to a department's offerings grows out of 
rational considerations within^ the academic concern of the department. The change 
in curricula represented by a new course is likely to be one of the most substan- 
tial Impacts of the innovative fund. In a thoughtful article comparing various 
curricula. Walker & Schaffarzick (1:974) conclude that > > dif f erent curricula are 
assoAated with different patter;a8 of achievemeiitT ^ whether a~aiff6f|^ pattern 
of achievement is better than^^he old pattern depends upon the conceptual systems 

/ . • , 

one employs to evaluate the , two patterns. The grant writer supplies a rationale 

/ » 

for why the new course he , is proposing should' be funded. The committee decides 
which of several proposals of this type should receive funding. It is up to the 
committee members to Provide a conceptual scheme wb^h will guide the overall 
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achievement pattern, they choose to fund acrosa the university. That conceptual 
scheme will establish the value of the achievement pattern resuitirig from funding 



• 10 

decisions ar^i ho^^fnlly wcn^ld ezxxrurage course devcJ^opmsnt proposals ctosoa&at 
vfch It. ,^rfbe erolo^iTO report aseocA^iec vtzh this type of denlopxnt fi&iald 
thus coiae frm tbe fcad cmrrt tte:€r si3 a^t fros tba individual greacee^. (Tixt 
overalf^r^>ort fl}^ by Kathleen Ilceaspki is descrlpt^v? and cot evaluative .) 
Id addition io the comltt'eas ur.lverslry vide pcrspectlxe, the departneat 
offering phe nev cotzrse 'd^: be asked to provide s^atioaale justifTing it, 
i.e., the depar rmen t could ca^Iain hew tie d^J||ibt of the coarse fits the 
strxicture of knovlfedge i£ the field of concern. Toe grantees' evalxsation 

T ^ 

function in this type of development is pri^rily an accounting role. Ee mast 
verify that the coarse ^^-ac de»?eloped and^taugfat and that the coney from the fend 
vas spent as the propo^fa^ said it vould be. Tne grantee is not in a .position 
tc establish ^he goodneas of his single course froa. vlthtn the franfiiwk of the 
cotirse it^elJ. Tnat type of evaluattoa mist be reserved for xrfthin course la- 
> provement efforts as previousiy discussed. 

Tne foregoing discassion leads to the 'conclusion that soae types of pro- 

j 

posals should not be subject tp the policy statement requiring clearly delineated 

plans for the evaluation of tht. project. If an evaluation is desir^ ^roa the 

» 

grantee vith a new course, it: can be -no laore then a critical case history of 
the course; the grantee should Bik infomed that tfals is all that is- expected. 
Providing the grantee vith h-^^-<j«lity (in the conEsittee's Judgxoent) exsople 

4 

case hiatories would help. In practice, cost grantees tollas^ed the case history 

J" • * 

approach when reporting back to the fund. Unfortunately the two /^r^n^t thcoes 

'\, * 
of tboae reports sees to be a r^etltion of the proposal PX ii a reliance on the 

ratings of selected students. Toe student ratings are prcAlcaatlc In thla caac 
* 

se of .the' dirpng poaalbility that the professors funded are oore effective . 
than norcal teachers (perhaps associated with the high interest in xeaehlng 

* ' 13 
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sbcnra by the great vriters; and because cost of the neu ccmrs-es ssrely gathered ^ 
fiCodeixcs already biased in f avor of the subject natter since they vere pre- 
dosiasatly voluntary enrolliaertt courses- *%lle conp arison df isaDrative 
courses vitn overall university ratings is of interest , this should be viesffed 
as veak evidence in favor of the funded changes. 

In the project entitled, Plannin? the Evaluation of Instructional Inno- 
vations, LLC proposed to accumlate data on ^ov effectively ZAC staff could 
vork vith innovative faculty, Cto research data is fo^s^d in the evaluation re- 
ports issuea by the center (See Gillncre S Sprenkle, 1975 Spreakle & Glllniore, 
1975, and Anderson, l975b, .for erainples) but ve vouid like to add a subjective 
tirpression. 5?c had 3 concern at the beginning of our evaluation efforts vlth 
the degree of recepti'^eness of the faculty tc evaluation, w erpected to encounter 
resistance, i.e., that p^sycholo-eical eopheidsp for 'he refuses tc play the game 
by Try rules/' In fact, the faculty neirbers vho have asked for our support have 
been very open to our advice and have in general been quite satisfied xd.th our 
assistance. Ha^i^er, in spi^e of encouragement from the fund coamlt-tee, not laany 

faculty vho were seeking funding vere anong those who consulted vith us. In 

t 

keying vith the distinction nsade earlier betveen grants seeking to inprove. 
existing courses and grants presenting new courses, the recoanendation is nade 
that grantees seeking tc improve the achicvezoent of students In existing courses 
should be required to present more rigorous plans for evaluation. In order to 
increase the standards of the judgiaental process for deterndning the adequacy of 
an evaluation section, ve recoaaend the addition of 'a faculty tiesbex trained in 
evaluation to the f^nd c oml ttee. Grant seekers with proposals for the lirprove- 
aent of existing courses should be required to consult vlth. the cortslttee eval- 
uator or with the staff at SAC. In viev of our experience thus far, the require- 

. 14 ' 
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ncsz ioT eyalg£tloa aLSSlsteace sixKJji a^t prove uadoly ocicroas to t±ie ff^lt? 
(t&ere arc, hcn^er, fim?Ie frelectlca effects In oar cj5>erlcac^^l^tcc all 
faculty dealt vltb to this point hsve been volGateers) . tSc see no need for 
rucfa cc?nsiiltatt03 for prop^osals establishing new courses • 

SmplrlcflLl Ees:ult6 

KfrdliB Developaaeat , There vcre nczzeross propo&alfi vhich asbe2 far lasncy for 
coarse- related nedta deveiopaier.t , but there vas very little data collected oa the 
effectiveness of the redla once it vas developed. Tnere can be no do:&t that 
e«dcnt8 can leam froa r^ella (e.g,, slides, tapes, films, etc,) btxt there la 
little evidence in the educational research literature to support the enthualafs 
of laany faculty for laedla over other laeans of inatmctica. 

At least on^ grantee. Professor Lataritren (Lattritzen 6 Daniels, 1975) » has 
reported on the effectiveness of adding laedia (in t^t fern of TV tapes) to his 
coarse (Engineering). In his Initial proposal. Professor Lauritren suggested 
that the TV tapes vonld be sotivatioQal. Two types of data vere cnalyxed to 
check that hypothesis: (1) students vere asked for estitfeates of the aoount of 
tiae they had studied for a unit of the coarse before b^lng given the unit cxaai- 
nation and (2) the perforr^ai^ce o^ the students on ti<e cxair^natioa . Coapariaoo 
of TV watchers vlth nonwatchers oa both cseasures failed to ^rxpport the hypo- 
thesis even tbou^ both ^wre plausible indicators of increased notivatlonal levela, 

A secoad professor (Ttoaas in EconosEics 200) has reported the ttse of a 
standardized econoslcs te^t fo( assessing student achievement. His students 
scored above the asan on this test and he attributes a portion of their success 
to hlijcxtensivc use of cedla. Ee also r^>orts diat tB? students gave hlgj^ 
ra^n^ to the aedia and Jhe course. Be took the liberty of sitting in Professor 
Tboaas' dass tor three of his lectures* Our observation convinced us that the 
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lattboi be uaed for &sseeslr:g jsbUs ef fectivcsess is Inralld. We hare no qxstrrel 
vitb the test he csed cvea choagh ve have vtot seea it; the test is tiost likely 
«a edequate aes^cre. Tae jrroiLeir is vith the faiiore to rule oo^t ?Tohsi>lt coa- 
foaadiiig vaz^iables. Srideace froa the edocatiqgal* research literatcre hat sbo«a* 
clarity of iajtroctioa to be ^positively related to stodeat echievcocat (See 
Eoecsthine & Purst, 1971, for ea sppro^rrlate review) • Dr. Thosas is o ?accei- 
leat lecturer; his verhel presenta-tioa has ell of the charecteristics of Lectczres 
leedlng to higher stodeat p>erforai£ac€. Is order for hia to claiiE the high 
echievezaeat of his stodects as oedi/i relaf^ed, Dr, Thonais voald have to gaji>er 
ha^eline data f ros his course vite^ the inedla. vcre not used for cocp ari»OQ vith 
his cour^ tshea the ^nedia vere p^eseaa. 

At an Innovative ?uad-EAC Anstroctioaal Ii g p r o v eae'^ t Scslnar , Dr . Tboiaas 
stated that a znajor reason for using nedia vas the cassive icattcnticn that he 
had observed In the large lecture classes in Kzte Hall and of fer^ the eorrea-^ 
pending hypothesis that aedia Inclusion vould .reduce the problen. Se agree tlxat 
inattention is a readily^ observable problem in Kane and that attention is a 
variable closely related to learning (Lof^, 1972; Horshj 1956; Blooa, 1974) • 
Dafortcnately Dr. Taosa&s did net syetenatically cfcscrve attentional brfiavior 
before and after the developaent of the laedia for Ecoaoaics 200, % dght also 
ask, '^ftxat is the student paying attention to?" One oovle vas prepared to show 
that joint ownership can lead to coaflicts. The aovle developed the these by 
showing tvo ''dudes" happily sharing a jointly purchased, piaea until a conflict 
over the last piece ended their beautiful relationship. We doubt that the point 
required the expense of a aovle to deaonstrate it. ^ After a stateaeat that kitteas 
are cute, color slides of kittcatf vere shown to illustrate the point. Several 
colc^r slides vere also shovn after a slallar statement about babies* Frov an 
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cstcrtalineat ?erfi:?erctlve tbt allies vcre veil doae, but the ettcattoa of tbc 
etodots vas oat necesetrlly dr«vn to tbe polat bdag laade froa o ecoDcnzic 
perspective nor ts It likely thrt tbe poisat vm isade In tbe o»t etto^Tirtcal 
issisaer cocscadt vlth ^tirfeat ediicrrefflcst. fccblevcagtt Is not tbe cJlj aeasure 
of l5i5>ort«nce in educaticaal settings, btjt St deserves ©ore sttestloa thsn it 
bts been getting. Ibe stndents' qtialitative ratings eboat bov tbe leartfing 
sitnatioa bas affected tbcc (student ratings data) are also iii;>ort.an^^t ve 
need to carefully r^^np tbe arg tfl &ent thax hl^b ratings and gjedia ^rgs^^ee-^ 
togetber. We believe tbat Dr. Thoisss vould deserve end votxld get higi quali- 
tative ratings y±th no core laeftis present than e blackboard or t)verbead pro- 
jector. , 

A nusber of rejtorts received fron grantees wltb ©edia related funding In- 
eluded Infonation abotxt 'effotts-..to^sess tbe quality of tbe developed rtedia. 
Hecall, bovevcr, tbat Eart^m^ definition -of value assessment requires tbat ve 
knov tbe conceptual scbeae used in valuation. Since none of tbe reports isade 
reference to achievement oeasures, tbe concQJtual scbeae underlying quality 
ju^gncnci^ould appear to involve suacb concepts m clarity of lasages, canera 
techniques, color vs. black and white, etc. In short the judgments are related 



to concepts of nedia professionalism audi like those the oDWEittec for selectlS? 
. the acadccy ssiaxda nigbt t^e. In spite of frequent assertiasis cade by facul^** ^ 
that they are coapetlng vltb television, t?e do not believe that the ata^wiards Ap- 
plied to television are ap pr o pr i ate for instructional oedia. Television seeks 
to entertain trbcreas Instructional oedia seeks to teach. Althou^ entertaining 
and teaching are not necessarily isutually exclusive, aedla developed for the tuo 
^ purposes tend to have cost differences. Entertaining usually ziox^ color pro- 
duction and other expensive eabelHshacat. Evidence collected In learning 
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rtudlei has shovn that ea Iscxpeasive black ezii \»4iit€ ,filii vith several liae 
drsvinge is as effective as tbc ssnie file vith zore elaborate dravliigs ^r^ ^ 
color pictography (Kay 6 Lamsdaiae, 1953). Eeoognltioa laeaory for pictures is 
as good for liae dreirljigs as it is for color photos of the same sceae 
(Bel&osy et al., 1974). A laajor jcstlficatioa for n^jg developaeat la fhat 
media can he used to free the instructors tline and that several iastroctors 
can use the sans iixedia. Tais is baaitraTly a cost related ar^xnieat aad the 
strength cf tae arguaeat is coasidetaoly^jjeaiiezted by including frills in taedla 
which are cere related to eatertaltnaeat than to instruction. Secondly, a cost- 
benefit anal ysis requires assessiaent of benefit, in this case assesenent of 
student achiev emen t . On til this achievement data is presented, the true cos^ 
of utedia cannot be detenzlned. 

rne addition of laedla to a course be researched in a msnner vhlch 
vlil allov evidence to be collected to address the questions ve have raised 
concerning laedla cost and me^dia effectiveness. Any future grants for Tf>f-dl^ 
^velopaent shcold iadude a detailed stateaent of the experimental procedure 
that vill be used, to provide evidence bearing on the i&edla* related issues 
raised in this -aect:^. ^ ^ 



Seed- Self -Paced Instruction. The personalised syst^ of instruction 
(?SI) propo^ by Keller (1968) has- the ^oUoving features: 

1. Students are alloved to proceed througii the course at their 
mn^ pace* 

^ Bach student mat reach an- 852 or 90X level of rsastery on 
/ a unit test before being alloved to. oovc to the next unit. 

, Repeated testing occurs until castery is attained* ^ 
3* Students are alloved to attend lectures and desaonst rations 
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as revard for finishing the £?? rp? r lat€ ualt* Lectures 
are oot seen as coarainlS^ critical iafonnatloa. 

4. i^i^r^ng occurs p^rliaarlly froa the vrlttea word. 

5. Proctors are used to test end consult vlth stodeaxt*. 

Khllc tb« overall ?SI plan has frequently been sbovn to be Krre effective in 
proaotlng student achievement than traditional lectures (See Shenaan^ 1974, 
for collected papers) » there has been little research to detemine which of the 
features is critical tc zi^j^^cess of the method. Instructors on this caapus 
I re codified the features, viiile, perhaps fals-ely, assurring they verc still 
getting the inprovciaent benefits advertised for ?SI* 

Engineering 14J, introductory PDRT2AJ< progranming, has been taught by f 
severe professors in a codified PSI fonsat. The students vere given frequent 
tests, they vere alloved one repeat on the test if they did not get 901 correct 
on the first test^ and learning vas primarily through the vritten vord. Tsw 
classes taught by this retbod vcre given the eaae final exadnation as five 
traditional classes (i»e., lectures, infrequent testing). There vas aD signi- 
f leant difference between the two types of classes (Anderson, 1975b). ?ro* 
feasor Dunn of xme of the sesd-PSI classes vas given a sisall grant to assist^ 
his in finishing a workbook containing three types of FOZZBAS practice probless. 
The sai^le sailtiple-choice probleias vere siollar to (and for 50Z of the pro- 
bias vere identical to) the cultlple-cholce probless used in unit ezafiitiatloai^. 
Students in a fall section of his cou»e received the voridbook vhlXe students in 
a vlnter section did not. The^e vas no difference, in the test perfonsance of 
the two g ro u ps even tbou^ the students rated the workbook favorably 
(Anderson, 1975b). 

By coc^aring the proportion of students getting an Itea correct across the 
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rasiy classes tested, we dlsccrreted that problems e^sy tor ozie class vere likely 

> 

be easy for other classes and questloas bard for oae class vere likely to be 
hard for the others (correlatioas 02 the order of ,70 to ,80 vere found). 55ie 
^ classes have concca teachiiig dlff teal ties, We tried to capitalize on this by 
giving stodents a vrltten statciaent of vaat they vere to ieam, a brief vrixten 
explanation for the statement, and a practice problcE for the stgtenaeat; this 
vas only dccte fcr one-half of the difficult test itcns (detendned froa a prior 
^ arter). The stadeats vere told that the handouts they vere receiving vere 
directly related. to difficult questions on the xxnit ptcttI nfttion» they vere 
a^lced to pay special attention tx> the handouts, Tne result vas a|aln no ia* 
provepent It achievement although the st;,^dcnts gave higji ratings to tfee presence 
of the handouts. 

^ Ton Love received a grant f rots the innovative fund for th^ purpose -of 

, , teaching structured prograansing concepts in the introdtictory PDRiaAS class. 

His students did not pexi^m aixj better on the final ezasdnation than the tradi- 
tional and sejsi-PSI^ classes. (His class did have less tiiae for the final 

exanination than other classes.) He did> bovever, collect data t&ich provide 

• .A 

a possible clue to the lack of success in these icp r o v ea ent efforts. He kept 
track of the g^ber of tioes students attez^ted to run their prograi&s before 
they vere able to run the program successfully. Frotc this data, I>6ve toocloded 
that the students vere requiring large aaounu of ^tlae in order to splve siople 
prograssing probles:s ^soae students took & 30 or 40 trials). It appears 
that the sain behavior of the^^tudents in the FQETS^ course is vritiag and . 
debugging progress (students say the course is quite tiae consusing)* Efforts 
at changing their learning throu^ vorkbooks and testing bava^ i^sibly been 
peri{>heral to vhat the student is actually doing. The progr aa a ^ g student is 
x»t, as is usual, studying for tests. 

ERIC .20 
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?TOgt£sndit% l9 an ideal place, tc study leamiag since tike studeats leeve 

observable tracl^ in their learning process. The error ruas.caa be cotryenlently 

> 

collected for analysts. With the data thus far collected aad further additix^ial 
effort focussing oa student progranalag efforts, it inigsrt he possible to produce 
an iiLpr o ver t en t in F0E12AS Instruction. 

Zxperlence vith the concentrated effort to tnprove the POKIEAS course in 
conbinatioa with ocoer experi^ence le^ads to cvo obser^rations, (1) hard vork by 
itself does not lead to iizprovenent and (2) repeated ^nprovemeat effort nsay be 
required before an inprovement project can shov positive results. Even ex- 
tended effort ussy not prove successful* Toese observations lead to a recoa- 
oendation to the fiznding cocndttee: the conalttee should be prepared to fund 
fe»er course inproveaent projects over a longer period of, tiae because « serious 
inprovcaent effort is likeiy to require isore than a one-ttoe infusion of funds 
into a course. This reconszendation does not^^s^ly to one-tloe grants fqz nev 
courses or achievesnent pattern developoent. / 

Hany instructors have reputed student ratings as part of the evaluation 
sent back to the innovative fund* While student ratings data is valuable, the 
in vhlch it is collected leaves nuch to be desired. Hatings froa a single 
course are noc an adequate ^urce of data since they offer nothing for conparl- 
son» Sed-PSI courses t^nd to receive good ratings but Sprenkle & Glllaore 
(1975) shoved that care laust be exercised in interpreting those ratings. Seven 
seai-PSI courses at IW vere rated more' hl^y than general univarsity courses 
using the saae evaluation fom, however, vhen ratings of'those seven courses 
vere compared vith ratings froa seven traditional sections of the sane cc^urses, . 
no differences vere found on Iteas coacon to both sets of evaluations, ikese . 
data, combined vith the hlg^b ratings given to the POSISAH voricbook and handouts 
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even tbaugb achievesiesnt lu ^ fCrw ' tf msat could net be attr&uted to ^he workbook aad 
haadoutfi, deadly point to the laade^oacy of the najcr forn of data (scodeat 
racl&gs) reported baci to the fxadlag cocEEittee. 

EvaluatioQ of Courses Desigoed to lafloeace Stadeat Attitude . LLC has 
been involved vita tvc coixrses^ttSf soagiit tc iafloeace students* attitudes • 
In one, student^ were given a scale laeasaring attitudes tov;3.rd disabled persoas 
(AID?) at the begiming and at the end of the course. There vas no difference 
between the pre and. post ATDP scores, analysis of the pre scores offers a 
reason for the lack of difference. Tne mean of the pre^course AID? scores vas 
125,37. This figure should be conpared vith aonLative data reported in Yuker, 
Block 6 Youang (1970) » the oean ATD? score for nales is 106.65 and for feaale^ 
is 114.18. A nonsative sazrple of disabled persons only had an average score 
*of 122 on the ATDP scale. The failure to produce change in certitudes is, due 
to the presence of an initial attitude ceiling effect. In the' second course, 
an atcenpt was tsade to favorably (concept not well defined by instructor) 
infltience student 3-ttitudes toward suicide (Gillmore & 5prenkle» 1975). Atti- 
tudes toward suicide were oeasured vith a Seraatit Differential technique 
* ^ whereby students responded to the word suicide with ratings on twcntypoae, 

seven point bi-polar scales, e/g., sad-happy, aggressive-oeek, brave-cowardly* 
The studfents showed no change in their ratings froa pre to post oeasures* 
There was no nomative data for the suicide scale as it was prepared solely for 
this class. It is worth noting that 67 of the 71 students answered yes to the 
question, 'Vhea enrolling, was this a course you wanted to take?" The lack of 
an attitude change in either course cay be due to strong subject selection ef- 
fects associated with volunteering for elective courses* It Is also, noteworthy 
that neither instructor oriMted their attitude change effort with a theory of 

^ on 
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attitude change. Evaljatton secti^s o£ prcpo&als seeklag fuading for attitude 
change efforts abould contain (1) provisions for pre-course sac post-course 
asa^saent of the attitude, (2) provlsioa for obtaining aorniative data froa 
sti^ats viio didn't voltnteer for the course, and (3) provision for e^cpLoriag 
the connection betveen the change effort and a theory of attitude change. 
Sunnsary of the jCecorruendations for Future Funding 

Ihe folloiriag njzbered statements are a sjcnsarization of recomendatlxms 
tsade-thus far. Toe carmrl ttee enould... 

1. •..categorize prc^&als into cvo types, (a) course iniprovea^t pro-- 
posals and (b) acnieveaent pattern proposals. Ihe comlttee should ask for 
rigorous evaluation of course 4jsprovesient efforts and collect case histories 
for the evaluation of achievesaent pattern proposals. 

( 2. ...consider providing grant seekers vith evaluation iafontation 
appropriate to the type of grant they are seeking. The coan»ittee TSitJn -educate 
the grant seekers vith respect to evaluation if the fund is to havens, strong 
evaluation prograa associated vith it. 

3. ...accept the responsibility for evaluating the overall pattern of 
achieveiaent they have funded. ^ ^ 

4. . ..add an evaluation specialist to the conniittee and ask grant seekers 
to consult vith the evaluator> 



5. ..-request nore rigorous valuation efforts froa faculty seeking 'ocdi*^ 
related grants. Student achievement sho^u^ be related to zaedia development. 
Additionally Dedia costs shotild be lovered by removing entertainaent-oriextted 
professional standards froo instructional media, 

6. «.*adopt the belief that course inproveaent is difficult to produce «nd 
document. In accordance vith this belief, course it^rovesent grants should cov»r 
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longer tiiae periods than acoievesaent pattern ^ants. 

7. . . . eacoora^e abandomaent of stixient ratings as the csajor type of 
reported data. Grantees should be encouraged to use achieveiDent data px^iaarlly 
and student ratings as secondary data. 

3. ...request before and after laeasures in csursfiS ja^ekiag to produce 
student attitude cnang^e. ifcnsative data frcm populations randomly selected 
snould also be requested tc control for the bias 4-^troduced by^tudents 
selecting the funded courses as electives. 

Ivaloation has been too long in cosiing to the individual instructor's 
classroom. If the fxmd cosnaittee encourages grant seekers to take self- 
evaluation^ seriously and if models and nethods can be given to the grantee, 
then icaybe evidence can be generated vhlch vill lead to and docxcaent real 
change for the better, dowever, the '^better" in the last sentence depends upon 
the rigor and general acceptability of the' conceptual analysis vhich leads tp 
it. The strength of the funds real in5>act on the learning of stxidents lies in 
the goodness of the conceptual base the fund cosndttee lays for evaluationi 
without such a base funded activity will be all effort and no 4cigroveiaent. 
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